Unsupervised deformable image registration method using cycle-consistent neural network and apparatus therefor

ABSTRACT

Disclosed are an unsupervised learning-based image registration method using a neural network with cycle consistency and an apparatus therefor. An image registration method includes receiving a first image and a second image for image registration, outputting a deformation field for the first image and the second image using an unsupervised learning-based neural network with cycle consistency for the deformation field, and generating a registration image for the first image and the second image based on a spatial deformation function using the output deformation field. The outputting of the deformation field includes outputting the deformation field for the first image for registering the first image to the second image may be output, when the first image is a moving image and the second image is a fixed image, and the generating of the registration image includes generating the registration image by applying the deformation field for the first image to the first image using the spatial deformation function.

CROSS-REFERENCE TO RELATED APPLICATIONS

A claim for priority under 35 U.S.C. § 119 is made to Korean PatentApplication No. 10-2020-0081299 filed on Jul. 2, 2020, in the KoreanIntellectual Property Office, the entire contents of which are herebyincorporated by reference.

BACKGROUND

Embodiments of the inventive concept described herein relate to anunsupervised learning-based image registration technology using a neuralnetwork with cycle consistency, and more particularly, to an imageregistration method capable of generating a registration image obtainedby registering a moving image and a fixed image using an unsupervisedlearning-based neural network with cycle consistency and an apparatustherefor.

Image registration is to transform two different images into onecoordinate system, and is used in various image processing fields suchas computer vision and medical images. In particular, registration ofmedical images is an essential step in locating and diagnosing aspecific lesion with respect to images taken over time. For example, inthe case of diagnosis of hepatocellular carcinoma (HCC), CT images aretaken at regular time intervals before and after infection of thecontrast agent, the contrast of the images is checked, the cancer isdiagnosed, and surgery or radiotherapy is planned. However, liver imagestaken at different phases are usually different in position and shape ofanatomical structures of the images due to patient motion, diseaseprogress, a patient's inhalation and exhalation, or the like. Therefore,image registration is very important to improve the accuracy ofdiagnosis and treatment.

The existing image registration method was developed to minimize anenergy function through an iterative method for a deformation space. Inparticular, the diffeomorphic image registration method has been muchresearched for the preservation of topology of an original image andone-to-one mapping between a fixed image and a moving image. However,these approaches usually require substantial time and extensivecomputation.

Recently, as AI-based techniques show high performance in the imageprocessing field, supervised learning/unsupervised learning-basedresearch using deep neural networks (or deep artificial neural networks)is also being conducted in the image registration field. Here, the deepneural network provides a deformation vector field capable of deformingan image, and the final deformed image is generated by a differentiableinterpolation technique such as a spatial transformer. Although theAI-based image registration technique has the advantage of having ashort image registration time, there is a disadvantage that theconstraint on the topology preservation of the original image is notclear. When the AI-based image registration technique is applied to animage with a large three-dimensional image size, such as a CT image, theregistration performance is limited.

SUMMARY

Embodiments of the inventive concept provide an image registrationmethod capable of generating a registration image obtained byregistering a moving image and a fixed image using an unsupervisedlearning-based neural network with cycle consistency and an apparatustherefor.

According to an exemplary embodiment, an image registration methodincludes receiving a first image and a second image for imageregistration, outputting a deformation field for the first image and thesecond image using an unsupervised learning-based neural network withcycle consistency for the deformation field, and generating aregistration image for the first image and the second image based on aspatial deformation function using the output deformation field.

The outputting of the deformation field may include outputting thedeformation field for the first image for registering the first imageand the second image may be output, when the first image is a movingimage and the second image is a fixed image, and the generating of theregistration image may include generating the registration image byapplying the deformation field for the first image to the first imageusing the spatial deformation function.

The neural network may include a first neural network that generates afirst registration image through the spatial deformation function byusing a third image and a fourth image as inputs and outputting adeformation field for the third image, a second neural network thatgenerates a second registration image through the spatial deformationfunction by using the third image and the fourth image as inputs andoutputting a deformation field for the fourth image, a third neuralnetwork that generates a third registration image through the spatialdeformation function by using the first registration image and thesecond registration image as inputs and outputting a deformation fieldfor the first registration image, and a fourth neural network thatgenerates a fourth registration image through the spatial deformationfunction by using the first registration image and the secondregistration image as inputs and outputting a deformation field for thesecond registration image.

The neural network may be trained in an unsupervised manner based oncyclic loss between the third image and the third registration image,cyclic loss between the fourth image and the fourth registration image,registration loss between the third image and the second registrationimage, registration loss between the fourth image and the firstregistration image, and identify loss between the third image and thefirst registration image or identify loss between the fourth image andthe second registration image when the third image is identical to thefourth image.

The neural network may be trained based on predefined cyclic loss,registration loss, and identity loss, with respect to a moving image, afixed image, and a registration image for the moving image and the fixedimage.

The neural network may include any one of a neural network based on aconvolution framelet and a neural network including a pooling layer andan unpooling layer.

According to an exemplary embodiment, an image registration methodincludes receiving a first medical image and a second medical image forimage registration, outputting a deformation field for the first medicalimage for registering the first medical image and the second medicalimage using an unsupervised learning-based neural network with cycleconsistency for the deformation field, and generating a registrationimage for the first image and the second image based on a spatialdeformation function using the output deformation field.

According to an exemplary embodiment, an image registration apparatusincludes a reception unit that receives a first image and a second imagefor image registration, and a registration unit that outputs adeformation field for a first image and a second image by using anunsupervised learning-based neural network with cycle consistency forthe deformation field, and generates a registration image for the firstimage and the second image based on a spatial deformation function usingthe output deformation field.

The registration unit may output the deformation field for the firstimage for registering the first image and the second image and generatethe registration image by applying the deformation field for the firstimage to the first image using the spatial deformation function, whenthe first image is a moving image and the second image is a fixed image.

The neural network may include a first neural network that generates afirst registration image through the spatial deformation function byusing a third image and a fourth image as inputs and outputting adeformation field for the third image, a second neural network thatgenerates a second registration image through the spatial deformationfunction by using the third image and the fourth image as inputs andoutputting a deformation field for the fourth image, a third neuralnetwork that generates a third registration image through the spatialdeformation function by using the first registration image and thesecond registration image as inputs and outputting a deformation fieldfor the first registration image, and a fourth neural network thatgenerates a fourth registration image through the spatial deformationfunction by using the first registration image and the secondregistration image as inputs and outputting a deformation field for thesecond registration image.

The neural network may be trained in an unsupervised manner based oncyclic loss between the third image and the third registration image,cyclic loss between the fourth image and the fourth registration image,registration loss between the third image and the second registrationimage, registration loss between the fourth image and the firstregistration image, and identify loss between the third image and thefirst registration image or identify loss between the fourth image andthe second registration image when the third image is identical to thefourth image. The neural network may be trained based on predefinedcyclic loss, registration loss, and identity loss, with respect to amoving image, a fixed image, and a registration image for the movingimage and the fixed image.

The neural network may include any one of a neural network based on aconvolution framelet and a neural network including a pooling layer andan unpooling layer.

BRIEF DESCRIPTION OF THE FIGURES

The above and other objects and features will become apparent from thefollowing description with reference to the following figures, whereinlike reference numerals refer to like parts throughout the variousfigures unless otherwise specified, and wherein:

FIG. 1 is an operation flowchart illustrating an image registrationmethod according to an embodiment of the inventive concept;

FIG. 2 shows an exemplary diagram of a framework for performing themethod of FIG. 1;

FIG. 3 shows an exemplary diagram for describing a loss for training aneural network used in the method of FIG. 1;

FIG. 4 shows an exemplary diagram of registration results and adeformation vector field according to the method of the inventiveconcept; and

FIG. 5 shows a configuration of an image registration apparatusaccording to an embodiment of the inventive concept.

DETAILED DESCRIPTION

Advantages and features of the inventive concept and methods forachieving them will be apparent with reference to embodiments describedbelow in detail in conjunction with the accompanying drawings. However,the inventive concept is not limited to the embodiments disclosed below,but can be implemented in various forms, and these embodiments are tomake the disclosure of the inventive concept complete, and are providedso that this disclosure will be thorough and complete and will fullyconvey the scope of the invention to those of ordinary skill in the art,which is to be defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the inventiveconcept. The singular expressions include plural expressions unless thecontext clearly dictates otherwise. In this specification, the terms“comprises” and/or “comprising” are intended to specify the presence ofstated features, integers, steps, operations, elements, parts orcombinations thereof, But do not preclude the presence or addition ofsteps, operations, elements, parts, or combinations thereof.

Unless defined otherwise, all terms (including technical and scientificterms) used herein have the same meanings as commonly understood by oneof ordinary skill in the art to which this invention belongs. Further,unless explicitly defined to the contrary, the terms defined in agenerally-used dictionary are not ideally or excessively interpreted.

Hereinafter, preferred embodiments of the inventive concept will bedescribed in detail with reference to the accompanying drawings. Thesame reference numerals are used for the same components in thedrawings, and duplicate descriptions of the same components are omitted.

Radiologists often diagnose the progress of disease by comparing medicalimages at different temporal phases. In case of diagnosis of liver tumorsuch as hepatocellular carcinoma (HCC), the contrast of normal livertissue and tumor region distinctly varies before and after the infectionof contrast agent. This provides radiologists an important clue todiagnose cancers and plan surgery or radiation therapy. However, liverimages taken at different phases are usually different in their shapedue to disease progress, breathing, patient motion, etc., so imageregistration is important to improve accuracy of dynamic studies.

Image registration methods according to embodiments are implemented in avariational framework that solves an energy minimization problem overthe space of deformations. Since the diffeomorphic image registrationensures the preservation of topology and one-to-one mapping between thesource and target images, the algorithmic extensions to largedeformation such as LDDMM and SyN have been applied to various imageregistration studies. However, these approaches usually requiresubstantial time and extensive computation.

To address this issue, recent image registration techniques are based ondeep neural networks that instantaneously generate a deformation field.In supervised learning approaches, the ground-truths of the deformationfield are required for training neural networks, which are typicallygenerated by the traditional registration method. However, theperformance of these existing supervised methods depends on the qualityof the ground-truth registration fields, or the existing supervisedmethods do not explicitly enforce the consistency criterion to uniquelydescribe the correspondences between two images.

Embodiments of the inventive concept are to generate a registrationimage in which a moving image is matched to a fixed image by using anunsupervised learning-based neural network with cycle consistency.

Here, the cycle consistency in the inventive concept can improvetopology preservation while generating fewer folding problems, and asingle neural network of the inventive concept provides deformableregistration between every pairs once the network is trained.

FIG. 1 is a flowchart of an image registration method according to anembodiment of the inventive concept.

Referring to FIG. 1, an image registration method according to anembodiment of the inventive concept may include receiving a first image,for example, a moving image, and a second image, for example, a fixedimage (S110), the first image and the second image being to beregistered.

When the first image and the second image are received in step S110, adeformation field for the first image and the second image is outputusing an unsupervised learning-based neural network with cycleconsistency for the deformation field (S120).

Here, in step S120, when the first image is a moving image and thesecond image is a fixed image, a deformation field for the first imageto register the first image and the second image may be output.

An image registration method of the inventive concept may be learned asan unsupervised learning model by being trained using neural networksincluding a first neural network that generates a first registrationimage through a spatial deformation function by using a third image anda fourth image included in the training data set as inputs andoutputting a deformation field for the third image, a second neuralnetwork that generates a second registration image through a spatialdeformation function by using the third image and the fourth image asinputs and outputting a deformation field for the fourth image, a thirdneural network that generates a third registration image through aspatial deformation function by using the first registration image andthe second registration image as inputs and outputting a deformationfield for the first registration image, and a fourth neural network thatgenerates a fourth registration image through a spatial deformationfunction by using the first registration image and the secondregistration image as inputs and outputting a deformation field for thesecond registration image.

In this case, the neural network may be trained in an unsupervisedmanner based on cyclic loss between the third image and the thirdregistration image, cyclic loss between the third image and the thirdregistration image, registration loss between the third image and thesecond registration image, registration loss between the fourth imageand the first registration image, and identify loss between the thirdimage and the first registration image or the fourth image and thesecond registration image when the third image is identical to thefourth image.

That is, the neural network of the inventive concept may be trainedbased on predefined cyclic loss, registration loss, and identity loss,with respect to a moving image and a fixed image, and a registrationimage for the moving image and the fixed image.

Furthermore, the neural network used in the inventive concept mayinclude not only a convolution framelet-based neural network, a neuralnetwork including a pooling layer and an unpooling layer, for example,U-Net, but also various types of neural networks applicable to theinventive concept.

A convolutional framelet refers to a method of representing an inputsignal through local and non-local bases. In order to reveal the blackbox characteristics of deep convolutional neural networks, a study on anew mathematical theory of deep convolutional framelets (Ye, J C., Han,Y., Cha, E.: Deep convolutional framelets: a general deep learningframework for inverse problems. SIAM Journal on Imaging Sciences 11(2),991-1048(2018)).

When the deformation field or deformation vector field of the firstimage is output in step S120, a registration image for the first imageand the second image is generated based on a spatial deformationfunction using the output deformation field (S130).

In this case, in step S130, the registration image may be generated byapplying a deformation field for the first image to the first imageusing a spatial deformation function.

The method of the inventive concept will be described with reference toFIGS. 2 to 4.

FIG. 2 shows an exemplary diagram of a framework for performing themethod of FIG. 1. As shown in FIG. 2, for the input images, A(P_(A)S_(A)) and B (P_(B)S_(B)), in different phases, two registrationnetworks are defined as G_(AB): (A,B)→φ_(AB) and G_(BA): (B,A)→φ_(BA),where φ_(AB) (resp. φ_(BA)) denotes the 3-D deformation field from A toB (resp. B to A). Here, P denotes the phase of the image, for example,the contrast of the image, and S denotes the shape of the image. A and Bshown in FIG. 2 may denote images with different phases and shapes. Ofcourse, in the inventive concept, various types of images such as imageshaving the same phase and different shapes and images having the sameshape and different phases may be used as inputs.

In the inventive concept, a 3D spatial transformation layer (or spatialtransformation function) T of the neural network is used to warp themoving image by the estimated deformation field, so that theregistration network is trained to minimize the dissimilarity betweenthe deformed moving source image and a fixed target image. Accordingly,once a pair of images is given to the registration network, the movingimages are deformed into fixed images.

To guarantee the topology preservation between the deformed and fixedimages, the cycle consistency constraint between the original movingimage and its re-deformed image may be adopted. That is, the deformedvolumes are given as the inputs to the networks again by switching theirorder to impose the cycle consistency. This constraint ensures that theshape of deformed images successively returns to the original shape.

The neural network shown in FIG. 2 may include a first neural network(G_(AB)) that generates a first registration image ({circumflex over(B)}) through a spatial deformation function by inputting an image “A”as a moving image and an image “B” as a fixed image and outputting adeformation field (φ_(AB)) for the image “A”, a second neural network(G_(BA)) that generates a second registration image (Â) through aspatial deformation function by inputting the image “B” as a movingimage and the image “A” as a fixed image and outputting a deformationfield (φ_(BA)) for the image “B”, a third neural network (G_(BA)) thatgenerates a third registration image (Ã) through a spatial deformationfunction by inputting the first registration image as a moving image andthe second registration image as a fixed image and outputting adeformation field ({circumflex over (φ)}_(BA)) for the firstregistration image and a fourth neural network (G_(AB)) that generates afourth registration image ({tilde over (B)}) through a spatialdeformation function by inputting the second registration image as amoving image and the first registration image as a fixed image andoutputting a deformation function ({circumflex over (φ)}_(AB)) for thesecond registration image. These two neural networks may be trained tomaximize similarity between the similarity between the deformed imageand the fixed image.

According to the inventive concept, it is possible to train a neuralnetwork by solving a loss function as shown in Equation 1 below.

=

_(regist) ^(AB)+

_(regist) ^(BA)+α

_(cycle)+β

_(identity)  [Equation 1]

where L_(regist), L_(cycle), and L_(identity) denote registration loss,cycle loss, and identity loss, respectively, and α and β denoteshyperparameters.

The method of the inventive concept may train a neural network in anunsupervised manner based on this loss function without ground-truthdeformation field.

That is, as shown in FIG. 3A, the neural network of the inventiveconcept may be trained in an unsupervised manner based on cycle lossbetween P_(A)S_(A) and the third registration image (P_(A)S_(A)) andbetween P_(B)S_(B) and the fourth registration image (P_(B)S_(B)),registration loss between P_(A)S_(A) and the second registration image(P_(B)S_(A)) and between the P_(B)S_(B) and the first registered image(P_(A)S_(B)) and identity loss between P_(A)S_(A) and P_(B)S_(B) orbetween P_(B)S_(B) and P_(A)S_(A), when two input images are the sameimage as shown in FIG. 3B. The cycle loss, the registration loss, andthe identity loss will be described in more detail as follows.

The registration loss function is based on the energy function ofclassical variational image registration. For example, the energyfunction for the registration of moving image A to the target volume Bis composed of two terms as in Equation 2:

_(regist) ^(AB)=

_(sim)(

(A,ϕ),B)+

_(reg)(ϕ)  [Equation 2]

where “A” is a moving image, “B” is a fixed image, and “T” denotes a 3Dspatial transformation function. L_(sim) computes image dissimilaritybetween the deformed image by the estimated deformation field p and thefixed image, and L_(reg) evaluates the smoothness of the deformationfield. In particular, the cross-correlation may be used as thesimilarity function to deal with the contrast change during CECT exam,and L2-loss may be used as a regularization function. Therefore, theregistration loss function may be written as in Equation 3 below.

_(regist) ^(AB)=−(

(A,ϕ _(AB))⊗B)+λ∥ϕ_(AB)∥₂  [Equation 3]

where ⊗ denotes the cross-correlation defined by Equation 4 below.

$\begin{matrix}{{x \otimes y} = \frac{{\langle {{x - \overset{\_}{x}},{y - \overset{\_}{y}}} \rangle }^{2}}{{{x - \overset{\_}{x}}}{{y - \overset{\_}{y}}}}} & \lbrack {{Equation}\mspace{20mu} 4} \rbrack\end{matrix}$

Here, X and Y denote the mean value of x and y, respectively.

The cycle consistency condition is implemented by minimizing the lossfunction as shown in FIG. 3A. Since an image A is first deformed to animage {circumflex over (B)} which is then deformed again by the othernetwork to generate an image Ã, the cyclic consistency imposes A≅Ã.Similarly, an image B should be successively deformed by the twonetworks to generate the image {tilde over (B)}. Then, the cycleconsistency imposes B≈{tilde over (B)}. As shown in FIG. 3A, since theneural network of the inventive concept receives both the moving imageand the fixed image, the implementation of cycle consistency loss may begiven by as the vector-form of cycle consistency condition as inEquation 5 below.

$\begin{matrix}{{\begin{bmatrix}A \\B\end{bmatrix} \simeq \begin{bmatrix}{\mathcal{T}( {\hat{B},{\hat{\phi}}_{BA}} )} \\{\mathcal{T}( {\overset{\hat{}}{A},{\hat{\phi}}_{A\; B}} )}\end{bmatrix}} = \begin{bmatrix}{\mathcal{T}( {{\mathcal{T}( {A,\phi_{AB}} )},{\hat{\phi}}_{BA}} )} \\{\mathcal{T}( {{\mathcal{T}( {B,\phi_{BA}} )},{\hat{\phi}}_{A\; B}} )}\end{bmatrix}} & \lbrack {{Equation}\mspace{20mu} 5} \rbrack\end{matrix}$

where ({circumflex over (B)}, Â):=(

(A,ϕ_(AB)),

(B,ϕ_(BA))). Thus, cycle consistency may be computed by Equation 6below.

$\begin{matrix}{\mathcal{L}_{cycle} = {{\begin{bmatrix}{\mathcal{T}( {\hat{B},{\hat{\phi}}_{BA}} )} \\{\mathcal{T}( {\overset{\hat{}}{A},{\hat{\phi}}_{A\; B}} )}\end{bmatrix} - \begin{bmatrix}A \\B\end{bmatrix}}}_{1}} & \lbrack {{Equation}\mspace{20mu} 6} \rbrack\end{matrix}$

where ∥⋅∥₁ denotes the l₁-norm.

Another important consideration for the loss function is that thenetwork should not change the stationary regions of the body. That is,the stationary regions should be the fixed points of the network. Asshown in FIG. 3B, this constraint can be implemented by imposing thatthe input image should not be changed when the identical images are usedas the floating and reference volume. More specifically, the inventiveconcept can use the identity loss as shown in Equation 7 below.

_(identity)=(

(A,G _(AB)(A,A))⊗A)−(

(B,G _(BA)(B,B))⊗(B)  [Equation 7]

By minimizing the identity loss of Equation 7, the cross-correlationbetween the deformed image and the fixed image can be maximized. Thus,the identity loss may guide the stability of the deformable fieldestimation in stationary regions.

The inventive concept can adopt VoxelMorph-1 as a baseline network togenerate a displacement vector field in width, height and depthdirections. The model of the inventive concept without both cycle andidentity losses may be equivalent to VoxelMorph-1. The 3D networkconsists of encoders, decoders, and skip connections similar to U-Net.

The 3D spatial transformation layer is to deform the moving volume withthe deformation field (p, and the spatial transformation function T maybe used with trilinear interpolation to warp the image A by ϕ, which canbe expressed as shown in Equation 8 below.

(A,ϕ)=

A(y)Π_(d∈{i,j,k})(1−|x _(d)+ϕ(x _(d))−y _(d)|)  [Equation 8]

where x denotes the voxel index, N(x+φ(x)) denotes an 8-voxel cubicneighborhood around x+φ(x), and d is three directions in 3D image space.

FIG. 4 shows an example of registration results (a) by the method of theinventive concept and the deformation vector field (b), and shows anexemplary view for multiphase 3D abdominal CT image registration.

As can be seen from FIG. 4, when image registration is performed by themethod of the inventive concept, image registration is possible within avery short time with similar registration accuracy compared to theconventional method, and has higher accuracy than the unsupervisedlearning-based image registration technique, which can be seen throughthe quantification index of the image deformation vector field. Inaddition, it can be seen that registration is possible for all imageshaving different image contrasts using a single neural network trainedby the method of the inventive concept, and its performance is also verygood.

As described above, the method according to an embodiment of theinventive concept may generate a registration image in which a movingimage is deformed into a fixed image by using an unsupervisedlearning-based neural network with cycle consistency.

The method according to an embodiment of the inventive concept canreduce the amount of computation and time for image registration, andcan also have high registration accuracy without loss of imageinformation.

The method according to an embodiment of the inventive concept canprovide an unsupervised learning-based image registration techniqueusing a neural network with cycle consistency, for example, aconvolutional neural network (CNN), and the cyclic constraint in theinventive concept is applied to the deformed image itself, thusincreasing the topology preservation performance of the original image(before deformation). In addition, the neural network in the inventiveconcept can be applied to various moving/fixed image domains such asmultiphase CT images. Accordingly, in a case where the neural networktraining is completed, image registration may be computed within a shorttime when the neural network receives any pair of new images.

In the field of computer vision, the method of the inventive concept maybe applied to transform, into one coordinate system, images obtained byphotographing one moving target through a fixed camera with a timedifference or images obtained by photographing one fixed target througha moving camera with a time difference. In the medical field, the methodof the inventive concept may be applied to images taken with a timedifference by various imaging devices, such as CT and MRI, as an imageregistration technique. Because the position and shape of anatomicalstructures in an image may vary depending on the patient's movement orthe progression of the lesion, the inventive concept may be applied toincrease the accuracy in planning disease diagnosis and treatmentthrough multiple images.

FIG. 5 illustrates a configuration of an image registration deviceaccording to an embodiment of the present disclosure, and illustrates aconceptual configuration of an apparatus for performing the methods ofFIGS. 1 to 4.

Referring to FIG. 5, an image registration device according to anembodiment of the inventive concept includes a reception unit 510 and aregistration unit 520. The reception unit 510 may include receiving afirst image for registration, for example, a moving image, and a secondimage, for example, a fixed image.

The registration unit 520 may output a deformation field for a firstimage and a second image by using an unsupervised learning-based neuralnetwork having cycle consistency for the deformation field, and generatea registration image for the first image and the second image based on aspatial deformation function using the output deformation field.

In this case, when the first image is a moving image and the secondimage is a fixed image, the registration unit 520 may output thedeformation field for the first image for registering the first imageand the second image and generate the registration image by applying thedeformation field for the first image to the first image using thespatial deformation function.

The neural networks may include a first neural network that generates afirst registration image through the spatial deformation function byusing a third image and a fourth image included in the training data setas inputs and outputting a deformation field for the third image, asecond neural network that generates a second registration image throughthe spatial deformation function by using the third image and the fourthimage as inputs and outputting a deformation field for the fourth image,a third neural network that generates a third registration image throughthe spatial deformation function by using the first registration imageand the second registration image as inputs and outputting a deformationfield for the first registration image, and a fourth neural network thatgenerates a fourth registration image through the spatial deformationfunction by using the first registration image and the secondregistration image as inputs and outputting a deformation field for thesecond registration image.

The neural network may be trained in an unsupervised manner based oncyclic loss between the third image and the third registration image,cyclic loss between the third image and the third registration image,registration loss between the third image and the second registrationimage, registration loss between the fourth image and the firstregistration image, and identify loss between the third image and thefirst registration image or the fourth image and the second registrationimage when the third image is identical to the fourth image.

The neural network may include any one of a neural network based on aconvolution framelet and a neural network including a pooling layer andan unpooling layer.

Although the description is omitted with reference to the apparatus ofFIG. 5, components constituting FIG. 5 may include all the contentsdescribed with reference to FIGS. 1 to 4, which are obvious to thoseskilled in the art.

The apparatus described herein may be implemented with hardwarecomponents and software components and/or a combination of the hardwarecomponents and the software components. For example, the apparatus andcomponents described in the embodiments may be implemented using one ormore general-purpose or special purpose computers, such as, for example,a processor, a controller and an arithmetic logic unit (ALU), a digitalsignal processor, a microcomputer, a field programmable array (FPA), aprogrammable logic unit (PLU), a microprocessor or any other devicecapable of executing and responding to instructions. The processingdevice may run an operating system (OS) and one or more softwareapplications that run on the OS. The processing device also may access,store, manipulate, process, and create data in response to execution ofthe software. For convenience of understanding, one processing device isdescribed as being used, but those skilled in the art will appreciatethat the processing device includes a plurality of processing elementsand/or multiple types of processing elements. For example, theprocessing device may include multiple processors or a single processorand a single controller. In addition, different processingconfigurations are possible, such a parallel processors.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, for independently orcollectively instructing or configuring the processing device to operateas desired. Software and/or data may be embodied in any type of machine,component, physical or virtual equipment, computer storage medium ordevice that is capable of providing instructions or data to or beinginterpreted by the processing device. The software also may bedistributed over network coupled computer systems so that the softwareis stored and executed in a distributed fashion. In particular, thesoftware and data may be stored by one or more computer readablerecording mediums.

The above-described methods may be embodied in the form of programinstructions that can be executed by various computer means and recordedon a computer-readable medium. The computer readable medium may includeprogram instructions, data files, data structures, and the like, aloneor in combination. Program instructions recorded on the media may bethose specially designed and constructed for the purposes of theinventive concept, or they may be of the kind well-known and availableto those having skill in the computer software arts. Examples ofcomputer readable recording media include magnetic media such as harddisks, floppy disks and magnetic tape, optical media such as CD-ROMs,DVDs, and magnetic disks such as floppy disks, Magneto-optical media,and hardware devices specifically configured to store and executeprogram instructions, such as ROM, RAM, flash memory, and the like.Examples of program instructions include not only machine code generatedby a compiler, but also high-level language code that can be executed bya computer using an interpreter or the like.

Although the embodiments have been described by the limited embodimentsand the drawings as described above, various modifications andvariations are possible to those skilled in the art from the abovedescription. For example, the described techniques may be performed in adifferent order than the described method, and/or components of thedescribed systems, structures, devices, circuits, etc. may be combinedor combined in a different form than the described method, or othercomponents, or even when replaced or substituted by equivalents, anappropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents tothe claims are within the scope of the following claims.

According to embodiments of the inventive concept, it is possible togenerate a registration image obtained by registering a moving image anda fixed image using an unsupervised learning-based neural network withcycle consistency.

According to embodiments of the inventive concept, it is possible toreduce the amount of computation and time for image registration, andalso have high registration accuracy without loss of image information.

In the field of computer vision, the inventive concept may be applied totransform, into one coordinate system, images obtained by photographingone moving target through a fixed camera with a time difference orimages obtained by photographing one fixed target through a movingcamera with a time difference. In the medical field, the method of theinventive concept may be applied to images taken with a time differenceby various imaging devices, such as CT and MRI, as an image registrationtechnique. Because the position and shape of anatomical structures in animage may vary depending on the patient's movement or the progression ofthe lesion, the inventive concept may be applied to increase theaccuracy in planning disease diagnosis and treatment through multipleimages.

While the inventive concept has been described with reference toexemplary embodiments, it will be apparent to those skilled in the artthat various changes and modifications may be made without departingfrom the spirit and scope of the inventive concept. Therefore, it shouldbe understood that the above embodiments are not limiting, butillustrative.

What is claimed is:
 1. An image registration method comprising:receiving a first image and a second image for image registration;outputting a deformation field for the first image and the second imageusing an unsupervised learning-based neural network with cycleconsistency for the deformation field; and generating a registrationimage for the first image and the second image based on a spatialdeformation function using the output deformation field.
 2. The imageregistration method of claim 1, wherein the outputting of thedeformation field includes outputting the deformation field for thefirst image for registering the first image and the second image may beoutput, when the first image is a moving image and the second image is afixed image, and wherein the generating of the registration imageincludes generating the registration image by applying the deformationfield for the first image to the first image using the spatialdeformation function.
 3. The image registration method of claim 1,wherein the neural network includes: a first neural network thatgenerates a first registration image through the spatial deformationfunction by using a third image and a fourth image as inputs andoutputting a deformation field for the third image, a second neuralnetwork that generates a second registration image through the spatialdeformation function by using the third image and the fourth image asinputs and outputting a deformation field for the fourth image, a thirdneural network that generates a third registration image through thespatial deformation function by using the first registration image andthe second registration image as inputs and outputting a deformationfield for the first registration image, and a fourth neural network thatgenerates a fourth registration image through the spatial deformationfunction by using the first registration image and the secondregistration image as inputs and outputting a deformation field for thesecond registration image.
 4. The image registration method of claim 3,wherein the neural network is trained in an unsupervised manner based oncyclic loss between the third image and the third registration image,cyclic loss between the fourth image and the fourth registration image,registration loss between the third image and the second registrationimage, registration loss between the fourth image and the firstregistration image, and identify loss between the third image and thefirst registration image or identify loss between the fourth image andthe second registration image when the third image is identical to thefourth image.
 5. The image registration method of claim 1, wherein theneural network is trained based on predefined cyclic loss, registrationloss, and identity loss, with respect to a moving image, a fixed image,and a registration image for the moving image and the fixed image. 6.The image registration method of claim 1, wherein the neural networkincludes any one of a neural network based on a convolution framelet anda neural network including a pooling layer and an unpooling layer.
 7. Animage registration method comprising: receiving a first medical imageand a second medical image for image registration; outputting adeformation field for the first medical image for registering the firstmedical image and the second medical image using an unsupervisedlearning-based neural network with cycle consistency for the deformationfield; and generating a registration image for the first image and thesecond image based on a spatial deformation function using the outputdeformation field.
 8. An image registration apparatus comprising: areception unit configured to receive a first image and a second imagefor image registration; and a registration unit configured to output adeformation field for a first image and a second image by using anunsupervised learning-based neural network with cycle consistency forthe deformation field, and generate a registration image for the firstimage and the second image based on a spatial deformation function usingthe output deformation field.
 9. The image registration apparatus ofclaim 8, wherein the registration unit outputs the deformation field forthe first image for registering the first image and the second image andgenerates the registration image by applying the deformation field forthe first image to the first image using the spatial deformationfunction, when the first image is a moving image and the second image isa fixed image.
 10. The image registration apparatus of claim 8, whereinthe neural network includes a first neural network that generates afirst registration image through the spatial deformation function byusing a third image and a fourth image as inputs and outputting adeformation field for the third image, a second neural network thatgenerates a second registration image through the spatial deformationfunction by using the third image and the fourth image as inputs andoutputting a deformation field for the fourth image, a third neuralnetwork that generates a third registration image through the spatialdeformation function by using the first registration image and thesecond registration image as inputs and outputting a deformation fieldfor the first registration image, and a fourth neural network thatgenerates a fourth registration image through the spatial deformationfunction by using the first registration image and the secondregistration image as inputs and outputting a deformation field for thesecond registration image.
 11. The image registration apparatus of claim10, wherein the neural network is trained in an unsupervised mannerbased on cyclic loss between the third image and the third registrationimage, cyclic loss between the fourth image and the fourth registrationimage, registration loss between the third image and the secondregistration image, registration loss between the fourth image and thefirst registration image, and identify loss between the third image andthe first registration image or identify loss between the fourth imageand the second registration image when the third image is identical tothe fourth image.
 12. The image registration apparatus of claim 8,wherein the neural network is trained based on predefined cyclic loss,registration loss, and identity loss, with respect to a moving image, afixed image, and a registration image for the moving image and the fixedimage.
 13. The image registration apparatus of claim 8, wherein theneural network includes any one of a neural network based on aconvolution framelet and a neural network including a pooling layer andan unpooling layer.